Field of the Invention
The present invention relates to an apparatus and method of augmenting video.
Description of the Prior Art
Augmented reality systems are known that use a video camera coupled to a computer to capture a video image, detect features of that captured video image, and then in response to the detected features, generate a graphical overlay to superpose on the captured video image which in some fashion augments the captured video image.
Referring to FIG. 1, in a typical augmented reality application a so-called fiduciary marker 800 of a known size is included in the scene to be captured by the video camera, in order to provide an easily detectable feature.
Fiduciary markers are typically high-contrast (e.g. black and white) and asymmetric patterns, so providing scope for pattern recognition in even relatively poor image capture conditions (e.g. with low resolution image capture, poor lighting, and the like). Such recognition typically provides an estimation of the position (x and y axis position) and rotation (by virtue of the marker's asymmetry) of the fiduciary marker within the captured video image.
Optionally in addition the distance of the fiduciary marker (its z-axis position) from the video camera may be estimated by comparing the size of the fiduciary marker in the captured image with a known size of the fiduciary marker (e.g. its size in pixels at a distance of 1 meter). Similarly, optionally a tilt of the fiduciary marker (i.e. its deviation from the x-y plane in the z direction) may be estimated by comparing the aspect ratio of the fiduciary marker in the captured image with the known aspect ratio of the fiduciary marker, and/or using other techniques such as detecting foreshortening (where a rectangular shape appears trapezoidal) or other distortions of the fiduciary marker in the captured video image.
The generated graphical overlay typically comprises a virtual graphics element that can then be superposed over the fiduciary marker, with the estimated position, rotation, distance, and tilt of the fiduciary marker used to modify the virtual graphics element as applicable.
The augmented image is then output to a display.
The subjective effect of this process is that the output video image comprises a graphical element (for example a monster or a castle) in place of the fiduciary marker, typically replicating the estimated position, rotation, distance and tilt of the fiduciary marker.
Alternatively, the fiduciary marker can be used in other ways. For example a graphical object can be positioned relative to the fiduciary marker but not covering it, or the fiduciary marker can be used to identify to an entertainment system or other computer a selected object or region of a scene; for example, placing a fiduciary marker on a table may cause an entertainment system to identify the table (for example by identifying a contiguous region of colour within a tolerance of the colour found near the fiduciary marker), after which the fiduciary marker can be removed.
However, it will be appreciated that in such augmented reality systems, the user's suspension of disbelief, which makes them complicit in the illusion that the graphical overlay is part of the captured scene, is at least in part dependent upon the reliability with which the graphical overlay matches or interacts with some real-world aspect of the scene. This reliability is frequently dependent upon estimates of the position and orientation of the fiduciary marker within the scene.
Thus it is desirable to reduce the occasions upon which such estimates cannot be reliably made.